{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "55c4401c-71f2-4cf2-a7c3-5cd490302e1b",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/joeferraratonic/llama_index/blob/tonic_validate_evaluator/docs/examples/evaluation/TonicValidateEvaluators.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8971d2b-ceda-48c7-8ebb-5d2b5039c8f9",
   "metadata": {},
   "source": [
    "# Tonic Validate Evaluators\n",
    "\n",
    "This notebook has some basic usage examples of how to use [Tonic Validate](https://github.com/TonicAI/tonic_validate)'s RAGs metrics using LlamaIndex. To use these evaluators, you need to have `tonic_validate` installed, which you can install via `pip install tonic-validate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04f90e80",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-evaluation-tonic-validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4da81670-8004-4124-b8f0-2db7decb19b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import pandas as pd\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n",
    "from llama_index.evaluation.tonic_validate import (\n",
    "    AnswerConsistencyEvaluator,\n",
    "    AnswerSimilarityEvaluator,\n",
    "    AugmentationAccuracyEvaluator,\n",
    "    AugmentationPrecisionEvaluator,\n",
    "    RetrievalPrecisionEvaluator,\n",
    "    TonicValidateEvaluator,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1cb065-ae86-4054-8a1e-ed3e1a58aa1c",
   "metadata": {},
   "source": [
    "## One Question Usage Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0033ce8-fc3d-4e15-8153-be529065a240",
   "metadata": {},
   "source": [
    "For this example, we have an example of a question with a reference correct answer that does not match the LLM response answer. There are two retrieved context chunks, of which one of them has the correct answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f1a592e-5c31-4132-964e-5d89bc7e46da",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"What makes Sam Altman a good founder?\"\n",
    "reference_answer = \"He is smart and has a great force of will.\"\n",
    "llm_answer = \"He is a good founder because he is smart.\"\n",
    "retrieved_context_list = [\n",
    "    \"Sam Altman is a good founder. He is very smart.\",\n",
    "    \"What makes Sam Altman such a good founder is his great force of will.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bebb6f7-244d-4e54-bb7c-a92148e4a0e2",
   "metadata": {},
   "source": [
    "The answer similarity score is a score between 0 and 5 that scores how well the LLM answer matches the reference answer. In this case, they do not match perfectly, so the answer similarity score is not a perfect 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33e75285-1924-4947-936b-11ba87a54036",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EvaluationResult(query='What makes Sam Altman a good founder?', contexts=['Sam Altman is a good founder. He is very smart.', 'What makes Sam Altman such a good founder is his great force of will.'], response='He is a good founder because he is smart.', passing=None, feedback=None, score=4.0, pairwise_source=None, invalid_result=False, invalid_reason=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_similarity_evaluator = AnswerSimilarityEvaluator()\n",
    "score = await answer_similarity_evaluator.aevaluate(\n",
    "    question,\n",
    "    llm_answer,\n",
    "    retrieved_context_list,\n",
    "    reference_response=reference_answer,\n",
    ")\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dd80d97-bbd5-44ed-adf4-34cb5f5ddcd3",
   "metadata": {},
   "source": [
    "The answer consistency score is between 0.0 and 1.0, and measure whether the answer has information that does not appear in the retrieved context. In this case, the answer does appear in the retrieved context, so the score is 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93a029a0-1799-427e-9a1d-1ff4503d9639",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EvaluationResult(query='What makes Sam Altman a good founder?', contexts=['Sam Altman is a good founder. He is very smart.', 'What makes Sam Altman such a good founder is his great force of will.'], response='He is a good founder because he is smart.', passing=None, feedback=None, score=1.0, pairwise_source=None, invalid_result=False, invalid_reason=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_consistency_evaluator = AnswerConsistencyEvaluator()\n",
    "\n",
    "score = await answer_consistency_evaluator.aevaluate(\n",
    "    question, llm_answer, retrieved_context_list\n",
    ")\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be605a98-8a53-49b0-af98-f045e1de017a",
   "metadata": {},
   "source": [
    "Augmentation accuracy measeures the percentage of the retrieved context that is in the answer. In this case, one of the retrieved contexts is in the answer, so this score is 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88c84e9f-aaf6-438e-aa63-c0578e3394fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EvaluationResult(query='What makes Sam Altman a good founder?', contexts=['Sam Altman is a good founder. He is very smart.', 'What makes Sam Altman such a good founder is his great force of will.'], response='He is a good founder because he is smart.', passing=None, feedback=None, score=0.5, pairwise_source=None, invalid_result=False, invalid_reason=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "augmentation_accuracy_evaluator = AugmentationAccuracyEvaluator()\n",
    "\n",
    "score = await augmentation_accuracy_evaluator.aevaluate(\n",
    "    question, llm_answer, retrieved_context_list\n",
    ")\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8222c185-c076-420f-b5c1-242bb6ac6c54",
   "metadata": {},
   "source": [
    "Augmentation precision measures whether the relevant retrieved context makes it into the answer. Both of the retrieved contexts are relevant, but only one makes it into the answer. For that reason, this score is 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2724130f-43f8-45b7-a48a-3b3cccde26a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EvaluationResult(query='What makes Sam Altman a good founder?', contexts=['Sam Altman is a good founder. He is very smart.', 'What makes Sam Altman such a good founder is his great force of will.'], response='He is a good founder because he is smart.', passing=None, feedback=None, score=0.5, pairwise_source=None, invalid_result=False, invalid_reason=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "augmentation_precision_evaluator = AugmentationPrecisionEvaluator()\n",
    "\n",
    "score = await augmentation_precision_evaluator.aevaluate(\n",
    "    question, llm_answer, retrieved_context_list\n",
    ")\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e021de2-0902-494d-9b97-7996533138ff",
   "metadata": {},
   "source": [
    "Retrieval precision measures the percentage of retrieved context is relevant to answer the question. In this case, both of the retrieved contexts are relevant to answer the question, so the score is 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d6c7b50-4670-4c2a-a37e-c341e882c406",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EvaluationResult(query='What makes Sam Altman a good founder?', contexts=['Sam Altman is a good founder. He is very smart.', 'What makes Sam Altman such a good founder is his great force of will.'], response='He is a good founder because he is smart.', passing=None, feedback=None, score=1.0, pairwise_source=None, invalid_result=False, invalid_reason=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieval_precision_evaluator = RetrievalPrecisionEvaluator()\n",
    "\n",
    "score = await retrieval_precision_evaluator.aevaluate(\n",
    "    question, llm_answer, retrieved_context_list\n",
    ")\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4faf4cd0-4e8f-4540-876a-bfdeefa7b241",
   "metadata": {},
   "source": [
    "The `TonicValidateEvaluator` can calculate all of Tonic Validate's metrics at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33bd6f82-0e70-4989-ade1-dab28c4cf12c",
   "metadata": {},
   "outputs": [],
   "source": [
    "tonic_validate_evaluator = TonicValidateEvaluator()\n",
    "\n",
    "scores = await tonic_validate_evaluator.aevaluate(\n",
    "    question,\n",
    "    llm_answer,\n",
    "    retrieved_context_list,\n",
    "    reference_response=reference_answer,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3da07443-8016-4787-b272-d0749e2ce945",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answer_consistency': 1.0,\n",
       " 'answer_similarity': 4.0,\n",
       " 'augmentation_accuracy': 0.5,\n",
       " 'augmentation_precision': 0.5,\n",
       " 'retrieval_precision': 1.0}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.score_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7d0476e-3468-434a-9fa8-860f952b85e8",
   "metadata": {},
   "source": [
    "You can also evaluate more than one query and response at once using `TonicValidateEvaluator`, and return a `tonic_validate` `Run` object that can be logged to the Tonic Validate UI (validate.tonic.ai).\n",
    "\n",
    "To do this, you put the questions, LLM answers, retrieved context lists, and reference answers into lists and cal `evaluate_run`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e2911ee-4fb4-48cb-bf16-ba3c8aa8cc1f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answer_consistency': 1.0,\n",
       " 'answer_similarity': 3.0,\n",
       " 'augmentation_accuracy': 0.5,\n",
       " 'augmentation_precision': 0.5,\n",
       " 'retrieval_precision': 1.0}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tonic_validate_evaluator = TonicValidateEvaluator()\n",
    "\n",
    "scores = await tonic_validate_evaluator.aevaluate_run(\n",
    "    [question], [llm_answer], [retrieved_context_list], [reference_answer]\n",
    ")\n",
    "scores.run_data[0].scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "965034c9-d9c4-42c8-9ef9-73ccbdb80a4d",
   "metadata": {},
   "source": [
    "## Labelled RAG Dataset Example\n",
    "\n",
    "Let's use the dataset `EvaluatingLlmSurveyPaperDataset` and evaluate the default LlamaIndex RAG system using Tonic Validate's answer similarity score. `EvaluatingLlmSurveyPaperDataset` is a `LabelledRagDataset`, so it contains reference correct answers for each question. The dataset contains 276 questions and reference answers about the paper *Evaluating Large Language Models: A Comprehensive Survey*.\n",
    "\n",
    "We'll use `TonicValidateEvaluator` with the answer similarity score metric to evaluate the responses from the default RAG system on this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b66c81cb-b6a4-45ac-b67c-e5fbc048c4e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  2.09it/s]\n",
      "Successfully downloaded EvaluatingLlmSurveyPaperDataset to ./data\n"
     ]
    }
   ],
   "source": [
    "!llamaindex-cli download-llamadataset EvaluatingLlmSurveyPaperDataset --download-dir ./data\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "\n",
    "from llama_index.core.llama_dataset import LabelledRagDataset\n",
    "\n",
    "from llama_index.core import VectorStoreIndex\n",
    "\n",
    "\n",
    "rag_dataset = LabelledRagDataset.from_json(\"./data/rag_dataset.json\")\n",
    "\n",
    "documents = SimpleDirectoryReader(input_dir=\"./data/source_files\").load_data(\n",
    "    num_workers=4\n",
    ")  # parallel loading\n",
    "\n",
    "index = VectorStoreIndex.from_documents(documents=documents)\n",
    "\n",
    "query_engine = index.as_query_engine()\n",
    "\n",
    "predictions_dataset = rag_dataset.make_predictions_with(query_engine)\n",
    "\n",
    "questions, retrieved_context_lists, reference_answers, llm_answers = zip(\n",
    "    *[\n",
    "        (e.query, e.reference_contexts, e.reference_answer, p.response)\n",
    "        for e, p in zip(rag_dataset.examples, predictions_dataset.predictions)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9be77d16-70db-4cc7-b7f8-5f93a024540b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tonic_validate.metrics import AnswerSimilarityMetric\n",
    "\n",
    "tonic_validate_evaluator = TonicValidateEvaluator(\n",
    "    metrics=[AnswerSimilarityMetric()], model_evaluator=\"gpt-4-1106-preview\"\n",
    ")\n",
    "\n",
    "scores = await tonic_validate_evaluator.aevaluate_run(\n",
    "    questions, retrieved_context_lists, reference_answers, llm_answers\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5077156-5e2b-40b9-97cb-bc25b7f5a734",
   "metadata": {},
   "source": [
    "The `overall_scores` gives the average score over the 276 questions in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41436382-effb-40f4-8be9-5314e67cf6be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answer_similarity': 2.2644927536231885}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.overall_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed68631-991b-4b92-ac4c-d6f2f0b7360e",
   "metadata": {},
   "source": [
    "Using `pandas` and `matplotlib`, we can plot a histogram of the similarity scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dd48f50-61d6-452c-9a28-b58fbdbf7a89",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGzCAYAAAAFROyYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAzW0lEQVR4nO3df3zN9f//8fvZbGfMfjdnlrGRGsqvYRYi9iYt8U5FVHj75F1NJf1483nnV8lQSTI/8i70Dguhd/ogrULlt/Tup1QjYVsqGyub7Pn9w2Xn27H52eE8p9v1cnldLs7z9TzP83i9zpnd93z9OA5jjBEAAIBF/HxdAAAAwIkIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgowHn23nvvyeFw6L333vPamHPmzJHD4dCuXbvcbR06dFCHDh289hqStGvXLjkcDs2ZM8er4+LsxcfHq3///r4uA7hgCCg4Z9OmTZPD4VBycrKvS/GJ0tJSvfzyy0pOTlZkZKRCQkJ0+eWX684779SGDRt8Xd5583//938aPXq018ctKSnRc889p2bNmik0NFTh4eFq1KiRBg0apC+//NLrr3e+TJo0SQ6HQ2+//fZJ+8yaNUsOh0P/+c9/LmBlf0xeXp4efvhhJSYmqlq1agoODlZSUpLGjh2rgwcP+ro8SdL8+fM1efJkX5cBL3HwXTw4V23atNG+ffu0a9cu7dy5U5dddpmvS7qgBg8erMzMTHXv3l0dO3ZUlSpVtGPHDq1YsUJ9+vRx/xIvLS1VSUmJAgMD5efnnb8Jjh07pqNHj8rpdMrhcEiSe/bEmzM1xhgVFxcrICBA/v7+kv7/dnv7v45u3bppxYoVuu2225SSkqKjR4/qyy+/1PLly/XEE09UmtmDffv2KS4uTv369dNLL71UYZ9rr71Wn3zyifbv36+AgIAzGjc+Pl4dOnTwyWzW5s2bdf311+vw4cO6/fbblZSUJEnasmWLsrKydPXVV+utt9664HWd6IYbbtCnn37qMbOISswA5+Dbb781ksySJUtMdHS0GT16tK9L8rpjx46ZX3/9tcJ1ubm5xuFwmLvuuqvcutLSUpOXl3e+yyunffv2pn379l4Z6+jRo6a4uLjCdenp6cbb/3Vs2rTJSDJPPvlkuXW//fabOXDggFdf71R+/fVXc+zYsT80RqdOnUxYWJg5cuRIuXXff/+98fPzM3ffffdZjVmnTh3Tr1+/P1TXufj555/NpZdealwul/niiy/Krc/NzTVPPPHEBa+rImlpaaZOnTq+LgNewiEenJN58+YpIiJCaWlpuvnmmzVv3rxyfcrOX3j66af1wgsvqF69enI6nWrZsqU2b97s0Tc3N1cDBgxQrVq15HQ6VbNmTXXv3t39l9DQoUMVFRXl8Vf7fffdJ4fDoSlTprjb8vLy5HA4NH36dHdbcXGxRo0apcsuu0xOp1NxcXF69NFHVVxc7FGDw+HQ4MGDNW/ePDVq1EhOp1MrV66scPtzcnJkjFGbNm3KrXM4HKpRo4b7cUXnoHTo0EFXXnml/vvf/6p9+/aqVq2aLrvsMi1evFiStGbNGiUnJ6tq1aq64ooryh0uqOgclBOVlJRo5MiRSkpKUlhYmIKDg9WuXTu9++67Hv1+/z5NnjzZ/T59/vnn5c5B6d+/vzIzM93bWbYYYxQfH6/u3buXq+PIkSMKCwvT3//+95PW+s0330hShfvT399fUVFRHm179+7VwIEDFRsbK6fTqYSEBN1zzz0qKSlx9/n22291yy23KDIyUtWqVVPr1q315ptveoxT9t5kZWXpscce06WXXqpq1aqpsLBQkrRx40Zdd911CgsLU7Vq1dS+fXt98MEHJ92OMrfffrsKCgrKvZ4kZWVlqbS0VH379pUkPf3007r66qsVFRWlqlWrKikpyf05OJXRo0e7Z89+72SfjRUrVqhdu3YKDg5WSEiI0tLS9Nlnn532dWbOnKm9e/dq0qRJSkxMLLfe5XLpscce82ibNm2a+2coNjZW6enp5Q4DneycmhPPpSp7jxYuXKgnn3xStWrVUlBQkDp16qSvv/7a43lvvvmmdu/e7f5cxsfHu9c///zzatSokapVq6aIiAi1aNFC8+fPP+32w4d8HJBQSSUmJpqBAwcaY4xZu3atkWQ2bdrk0ScnJ8dIMs2aNTOXXXaZmTBhgpk4caK55JJLTK1atUxJSYm779VXX23CwsLMY489Zv71r3+ZcePGmWuvvdasWbPGGGPMkiVLjCTzySefuJ/TpEkT4+fnZ26++WZ326JFi4wk8+mnnxpjjs+CdO7c2VSrVs0MGTLEzJw50wwePNhUqVLFdO/e3aNeSaZBgwYmOjrajBkzxmRmZpqPPvqowu3ft2+fkWTS0tJMUVHRKffVu+++aySZd999193Wvn17Exsba+Li4swjjzxinn/+edOwYUPj7+9vsrKyTExMjBk9erSZPHmyufTSS01YWJgpLCx0P3/27NlGksnJyfEY8/czKD/88IOpWbOmGTp0qJk+fbqZOHGiueKKK0xAQIDHdpW9Tw0bNjR169Y148ePN88++6zZvXu3e93s2bONMcZ8+OGH5i9/+YuRZP7973+7F2OM+ec//2kCAgLMjz/+6LH9CxcuNJLM2rVrT7qPPvzwQyPJ3HXXXebo0aOn3J979+41sbGx7vd0xowZZsSIEaZBgwbm559/NsYc/6ve5XKZkJAQ889//tNMmjTJ/XlZsmRJufemYcOGpmnTpmbSpEkmIyPDFBUVmezsbBMYGGhSUlLMM888Y5599lnTuHFjExgYaDZu3HjKGgsKCkxQUJDp2bNnuXXNmzc3derUMaWlpcYYY2rVqmXuvfdeM3XqVDNp0iTTqlUrI8ksX77c43knzqCMGjWqwpmsij4bL7/8snE4HOa6664zzz//vJkwYYKJj4834eHhHv0qcvXVV5uqVauedEbtRGV1paammueff94MHjzY+Pv7m5YtW3r8zJ9sRujEz3HZe9SsWTOTlJRknn32WTN69GhTrVo106pVK3e/t956yzRt2tRccskl7s/l0qVLjTHGvPDCC0aSufnmm83MmTPNc889ZwYOHGjuv//+M9om+AYBBWdty5YtRpJZvXq1Meb4IY1atWqZBx54wKNf2S+3qKgo89NPP7nbX3/9dSPJvPHGG8aY41PIksxTTz110tfMz883ksy0adOMMcYcPHjQ+Pn5mVtuucW4XC53v/vvv99ERka6//P/97//bfz8/My6des8xpsxY4aRZD744AN3myTj5+dnPvvsszPaD3feeaeRZCIiIsxf//pX8/TTT1c4BX6ygCLJzJ8/39325ZdfumvYsGGDu33VqlUeIcGYMwsov/32W7lfKj///LNxuVzmb3/7m7ut7H0KDQ01+fn5Hv1PDCjGnPwQz44dO4wkM336dI/2G2+80cTHx7vfk4qUlpa694nL5TK33XabyczMNLt37y7X98477zR+fn5m8+bNFY5jjDFDhgwxkjze90OHDpmEhAQTHx/vPoRT9t7UrVvX/PLLLx7j1K9f33Tp0sWj7l9++cUkJCSYv/zlLyfdljK33HKLCQoKMgUFBe62svd4+PDhHmP+XklJibnyyitNx44dPdrPNaAcOnTIhIeHlzscmZuba8LCwio8TPl7ERERpkmTJqfsUyY/P98EBgaazp07exwmmzp1qpFkXnrppZNuT5mTBZQGDRp4fJ6fe+65cn+0nOwQT/fu3U2jRo3OaBtgDw7x4KzNmzdPLpdL1157raTjU/29evVSVlaWjh07Vq5/r169FBER4X7crl07Scen4CWpatWqCgwM1Hvvvaeff/65wteMjo5WYmKi1q5dK0n64IMP5O/vr0ceeUR5eXnauXOnJGndunVq27ate+p70aJFatCggRITE3XgwAH30rFjR0kqd7ijffv2atiw4Rnth9mzZ2vq1KlKSEjQ0qVL9fDDD6tBgwbq1KmT9u7de9rnV69eXb1793Y/vuKKKxQeHq4GDRp4XBlV9u+y/XWm/P39FRgYKOn4ibo//fSTfvvtN7Vo0ULbtm0r179nz56Kjo4+q9f4vcsvv1zJyckeh/t++uknrVixQn379q3wcEQZh8OhVatWaezYsYqIiNCCBQuUnp6uOnXqqFevXu7DA6WlpVq2bJm6deumFi1aVDiOdPxKo1atWqlt27buddWrV9egQYO0a9cuff755x7P69evn6pWrep+vH37du3cuVN9+vTRjz/+6P7cFBUVqVOnTlq7dq1KS0tPuT9uv/12HTlyREuWLHG3lR1SKDu8I8njdX/++WcVFBSoXbt2Fb5H52L16tU6ePCgbrvtNo+fAX9/fyUnJ5f7GThRYWGhQkJCzui13n77bZWUlGjIkCEeJ4TfddddCg0NrfCQ15kaMGCA+/Mslf9/5FTCw8P1/ffflzu0DLsRUHBWjh07pqysLF177bXKycnR119/ra+//lrJycnKy8tTdnZ2uefUrl3b43FZWCkLI06nUxMmTNCKFSvkcrl0zTXXaOLEicrNzfV4Xrt27bRu3TpJx4NIixYt1KJFC0VGRmrdunUqLCzUxx9/7P6PS5J27typzz77TNHR0R7L5ZdfLknKz8/3eI2EhIQz3hd+fn5KT0/X1q1bdeDAAb3++uvq2rWr3nnnHY/gcTK1atUq90s7LCxMcXFx5doknTS8ncrcuXPVuHFjBQUFKSoqStHR0XrzzTdVUFBQru/ZbPvJ3Hnnnfrggw+0e/duSccD4tGjR3XHHXec9rlOp1P//Oc/9cUXX2jfvn1asGCBWrdurYULF2rw4MGSpB9++EGFhYW68sorTznW7t27dcUVV5Rrb9CggXv975247WWBt1+/fuU+O//6179UXFxc4T78va5duyoyMtLjPIcFCxaoSZMmatSokbtt+fLlat26tYKCghQZGano6GhNnz79tOOfqbJt6dixY7lteeutt8r9DJwoNDRUhw4dOqPXKtuvJ+77wMBA1a1bt9x+Pxun+3/kVP7xj3+oevXqatWqlerXr6/09PQzOpcIvlXF1wWgcnnnnXe0f/9+ZWVlKSsrq9z6efPmqXPnzh5tZZennsj87oTXIUOGqFu3blq2bJlWrVqlESNGKCMjQ++8846aNWsmSWrbtq1mzZqlb7/9VuvWrVO7du3kcDjUtm1brVu3TrGxsSotLfUIKKWlpbrqqqs0adKkCms4MQz8/q/ZsxEVFaUbb7xRN954ozp06KA1a9Zo9+7dqlOnzkmfc7L9cib760y88sor6t+/v3r06KFHHnlENWrUkL+/vzIyMtwnpf7euW777/Xu3VsPPvig5s2bp//93//VK6+8ohYtWlQYFk6lZs2a6t27t3r27KlGjRpp4cKF5/Xy2hO3vWx25KmnnlLTpk0rfE716tVPOWZAQIBuvfVWzZo1S3l5efruu++0c+dOTZw40d1n3bp1uvHGG3XNNddo2rRpqlmzpgICAjR79uzTnsB5shmpE2cxy7bl3//+t2JiYsr1r1Ll1L8GEhMTtX37dvel8t5yqvor+hn4Iz8XDRo00I4dO7R8+XKtXLlSr732mqZNm6aRI0dqzJgxZ1c4LhgCCs7KvHnzVKNGDfeVHL+3ZMkSLV26VDNmzDinX3b16tXTQw89pIceekg7d+5U06ZN9cwzz+iVV16R9P+ndFevXq3Nmzdr2LBhkqRrrrlG06dPV2xsrPvmUb8f8+OPP1anTp1OeYjBm1q0aKE1a9Zo//79pwwo59vixYtVt25dLVmyxGPbR40a9YfGPdV+jIyMVFpamubNm6e+ffvqgw8++EM3zgoICFDjxo21c+dOHThwQDVq1FBoaKg+/fTTUz6vTp062rFjR7n2shu+ne59qVevnqTjswepqannWP3xQzkzZszQq6++qpycHDkcDt12223u9a+99pqCgoK0atUqOZ1Od/vs2bNPO3bZDMLBgwcVHh7ubj9xlqJsW2rUqHFO29KtWzetX79er732mkftFSnbrzt27FDdunXd7SUlJcrJyfF4/YiIiApv8LZ7926P556NU302g4OD1atXL/Xq1UslJSW66aab9OSTT2r48OEKCgo6p9fD+cUhHpyxX3/9VUuWLNENN9ygm2++udwyePBgHTp06KzvjvnLL7/oyJEjHm316tVTSEiIx6XACQkJuvTSS/Xss8/q6NGj7ktS27Vrp2+++UaLFy9W69atPf4ivPXWW7V3717NmjWrwu0pKio6q1rL5ObmljuPQTr+H3F2drb8/Px8fuO6sr84f/8X5saNG7V+/fo/NG5wcLAknfTuoXfccYc+//xzPfLII/L39z+jw107d+7Ud999V6794MGDWr9+vSIiIhQdHS0/Pz/16NFDb7zxhrZs2VKuf9m2Xn/99dq0aZPHthYVFemFF15QfHz8ac8zSkpKUr169fT000/r8OHD5db/8MMPp90m6fhl0/Hx8XrllVf06quvqn379qpVq5Z7vb+/vxwOh8esx65du7Rs2bLTjl0WPMrOy5KOb+PcuXM9+nXp0kWhoaEaN26cjh49etbbcvfdd6tmzZp66KGH9NVXX5Vbn5+fr7Fjx0qSUlNTFRgYqClTpnh87l588UUVFBQoLS3No/4NGzZ4XBq+fPly7dmz55T1nEpwcHCFh8Z+/PFHj8eBgYFq2LChjDEV7hPYgRkUnLH//Oc/OnTokG688cYK17du3VrR0dGaN2+eevXqdcbjfvXVV+rUqZNuvfVWNWzYUFWqVNHSpUuVl5dX7pdbu3btlJWVpauuusr9F2Tz5s0VHBysr776Sn369PHof8cdd2jhwoW6++679e6776pNmzY6duyYvvzySy1cuFCrVq2q8GTL0/n+++/VqlUrdezYUZ06dVJMTIzy8/O1YMECffzxxxoyZIguueSSsx7Xm2644QYtWbJEf/3rX5WWlqacnBzNmDFDDRs2rPCX7pkqm6G6//771aVLl3IhJC0tTVFRUVq0aJG6du3qcU+Yk/n444/Vp08fde3aVe3atVNkZKT27t2ruXPnat++fZo8ebI7cI0bN05vvfWW2rdvr0GDBqlBgwbav3+/Fi1apPfff1/h4eEaNmyYFixYoK5du+r+++9XZGSk5s6dq5ycHL322munvaOvn5+f/vWvf6lr165q1KiRBgwYoEsvvVR79+7Vu+++q9DQUL3xxhun3S6Hw6E+ffpo3LhxkqTHH3/cY31aWpomTZqk6667Tn369FF+fr4yMzN12WWX6b///e8px+7cubNq166tgQMHusPgSy+9pOjoaI+wFxoaqunTp+uOO+5Q8+bN1bt3b3efN998U23atNHUqVNP+joRERFaunSprr/+ejVt2tTjTrLbtm3TggULlJKSIun4yezDhw/XmDFjdN111+nGG2/Ujh07NG3aNLVs2VK33367e9z/+Z//0eLFi3Xdddfp1ltv1TfffKNXXnnFHbzORVJSkl599VUNHTpULVu2VPXq1dWtWzd17txZMTExatOmjVwul7744gtNnTpVaWlpZ3wCMHzAh1cQoZLp1q2bCQoKOuV9P/r3728CAgLMgQMH3JeoVnT5sCQzatQoY4wxBw4cMOnp6SYxMdEEBwebsLAwk5ycbBYuXFjueZmZmUaSueeeezzaU1NTjSSTnZ1d7jklJSVmwoQJplGjRsbpdJqIiAiTlJRkxowZ43EJqCSTnp5+RvuisLDQPPfcc6ZLly6mVq1aJiAgwISEhJiUlBQza9Ysj0tTT3aZcUWXPdapU8ekpaWVaz+xtjO5zLi0tNSMGzfO1KlTxzidTtOsWTOzfPly069fP49LMU/1PlV0mfFvv/1m7rvvPhMdHW0cDkeFl7ree++95S6jPpW8vDwzfvx40759e1OzZk1TpUoVExERYTp27GgWL15crv/u3bvNnXfeaaKjo43T6TR169Y16enpHpehfvPNN+bmm2824eHhJigoyLRq1arcvUXK3ptFixZVWNdHH31kbrrpJhMVFWWcTqepU6eOufXWWyv8nJ3MZ599ZiQZp9Ppvk/L77344oumfv36xul0msTERDN79uwKLyGu6LLcrVu3muTkZBMYGGhq165tJk2aVOFno2xbu3TpYsLCwkxQUJCpV6+e6d+/v9myZcsZbce+ffvMgw8+aC6//HITFBRkqlWrZpKSksyTTz7p8XNkzPHLihMTE01AQIBxuVzmnnvuqXDbn3nmGXPppZcap9Np2rRpY7Zs2XLSy4xPfI8q+mwePnzY9OnTx4SHhxtJ7s/5zJkzzTXXXON+H+vVq2ceeeSRcnXDLnwXDwCve/DBB/Xiiy8qNzdX1apV83U5ACohzkEB4FVHjhzRK6+8op49exJOAJwzzkEB4BX5+fl6++23tXjxYv3444964IEHfF0SgEqMgALAKz7//HP17dtXNWrU0JQpU056/xAAOBOcgwIAAKzDOSgAAMA6BBQAAGCdSnkOSmlpqfbt26eQkJALdvtyAADwxxhjdOjQIcXGxp72homVMqDs27ev3Je8AQCAymHPnj0eX/tQkUoZUMpuTbxnzx6Fhob6uBoAAHAmCgsLFRcXd0ZfMVApA0rZYZ3Q0FACCgAAlcyZnJ7BSbIAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYJ2zDihr165Vt27dFBsbK4fDoWXLlnmsN8Zo5MiRqlmzpqpWrarU1FTt3LnTo89PP/2kvn37KjQ0VOHh4Ro4cKAOHz78hzYEAABcPM46oBQVFalJkybKzMyscP3EiRM1ZcoUzZgxQxs3blRwcLC6dOmiI0eOuPv07dtXn332mVavXq3ly5dr7dq1GjRo0LlvBQAAuKg4jDHmnJ/scGjp0qXq0aOHpOOzJ7GxsXrooYf08MMPS5IKCgrkcrk0Z84c9e7dW1988YUaNmyozZs3q0WLFpKklStX6vrrr9f333+v2NjY075uYWGhwsLCVFBQwJcFAgBQSZzN72+vnoOSk5Oj3NxcpaamutvCwsKUnJys9evXS5LWr1+v8PBwdziRpNTUVPn5+Wnjxo0VjltcXKzCwkKPBQAAXLyqeHOw3NxcSZLL5fJod7lc7nW5ubmqUaOGZxFVqigyMtLd50QZGRkaM2aMN0s9pfhhb16w17LJrvFpvi4BAABJleQqnuHDh6ugoMC97Nmzx9clAQCA88irASUmJkaSlJeX59Gel5fnXhcTE6P8/HyP9b/99pt++uknd58TOZ1OhYaGeiwAAODi5dWAkpCQoJiYGGVnZ7vbCgsLtXHjRqWkpEiSUlJSdPDgQW3dutXd55133lFpaamSk5O9WQ4AAKikzvoclMOHD+vrr792P87JydH27dsVGRmp2rVra8iQIRo7dqzq16+vhIQEjRgxQrGxse4rfRo0aKDrrrtOd911l2bMmKGjR49q8ODB6t279xldwQMAAC5+Zx1QtmzZomuvvdb9eOjQoZKkfv36ac6cOXr00UdVVFSkQYMG6eDBg2rbtq1WrlypoKAg93PmzZunwYMHq1OnTvLz81PPnj01ZcoUL2wOAAC4GPyh+6D4yvm+DwpX8QAA4H0+uw8KAACANxBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdrweUY8eOacSIEUpISFDVqlVVr149PfHEEzLGuPsYYzRy5EjVrFlTVatWVWpqqnbu3OntUgAAQCXl9YAyYcIETZ8+XVOnTtUXX3yhCRMmaOLEiXr++efdfSZOnKgpU6ZoxowZ2rhxo4KDg9WlSxcdOXLE2+UAAIBKqIq3B/zwww/VvXt3paWlSZLi4+O1YMECbdq0SdLx2ZPJkyfrscceU/fu3SVJL7/8slwul5YtW6bevXt7uyQAAFDJeH0G5eqrr1Z2dra++uorSdLHH3+s999/X127dpUk5eTkKDc3V6mpqe7nhIWFKTk5WevXr69wzOLiYhUWFnosAADg4uX1GZRhw4apsLBQiYmJ8vf317Fjx/Tkk0+qb9++kqTc3FxJksvl8niey+VyrztRRkaGxowZ4+1SAQCApbw+g7Jw4ULNmzdP8+fP17Zt2zR37lw9/fTTmjt37jmPOXz4cBUUFLiXPXv2eLFiAABgG6/PoDzyyCMaNmyY+1ySq666Srt371ZGRob69eunmJgYSVJeXp5q1qzpfl5eXp6aNm1a4ZhOp1NOp9PbpQIAAEt5fQbll19+kZ+f57D+/v4qLS2VJCUkJCgmJkbZ2dnu9YWFhdq4caNSUlK8XQ4AAKiEvD6D0q1bNz355JOqXbu2GjVqpI8++kiTJk3S3/72N0mSw+HQkCFDNHbsWNWvX18JCQkaMWKEYmNj1aNHD2+XAwAAKiGvB5Tnn39eI0aM0L333qv8/HzFxsbq73//u0aOHOnu8+ijj6qoqEiDBg3SwYMH1bZtW61cuVJBQUHeLgcAAFRCDvP7W7xWEoWFhQoLC1NBQYFCQ0O9Pn78sDe9PmZlsGt8mq9LAABcxM7m9zffxQMAAKxDQAEAANYhoAAAAOt4/SRZAAAuFn/WcxIl35+XyAwKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA65yWg7N27V7fffruioqJUtWpVXXXVVdqyZYt7vTFGI0eOVM2aNVW1alWlpqZq586d56MUAABQCXk9oPz8889q06aNAgICtGLFCn3++ed65plnFBER4e4zceJETZkyRTNmzNDGjRsVHBysLl266MiRI94uBwAAVEJVvD3ghAkTFBcXp9mzZ7vbEhIS3P82xmjy5Ml67LHH1L17d0nSyy+/LJfLpWXLlql3797eLgkAAFQyXp9B+c9//qMWLVrolltuUY0aNdSsWTPNmjXLvT4nJ0e5ublKTU11t4WFhSk5OVnr16+vcMzi4mIVFhZ6LAAA4OLl9YDy7bffavr06apfv75WrVqle+65R/fff7/mzp0rScrNzZUkuVwuj+e5XC73uhNlZGQoLCzMvcTFxXm7bAAAYBGvB5TS0lI1b95c48aNU7NmzTRo0CDdddddmjFjxjmPOXz4cBUUFLiXPXv2eLFiAABgG68HlJo1a6phw4YebQ0aNNB3330nSYqJiZEk5eXlefTJy8tzrzuR0+lUaGioxwIAAC5eXg8obdq00Y4dOzzavvrqK9WpU0fS8RNmY2JilJ2d7V5fWFiojRs3KiUlxdvlAACASsjrV/E8+OCDuvrqqzVu3Djdeuut2rRpk1544QW98MILkiSHw6EhQ4Zo7Nixql+/vhISEjRixAjFxsaqR48e3i4HAABUQl4PKC1bttTSpUs1fPhwPf7440pISNDkyZPVt29fd59HH31URUVFGjRokA4ePKi2bdtq5cqVCgoK8nY5AACgEvJ6QJGkG264QTfccMNJ1zscDj3++ON6/PHHz8fLAwCASo7v4gEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOuc9oIwfP14Oh0NDhgxxtx05ckTp6emKiopS9erV1bNnT+Xl5Z3vUgAAQCVxXgPK5s2bNXPmTDVu3Nij/cEHH9Qbb7yhRYsWac2aNdq3b59uuumm81kKAACoRM5bQDl8+LD69u2rWbNmKSIiwt1eUFCgF198UZMmTVLHjh2VlJSk2bNn68MPP9SGDRsqHKu4uFiFhYUeCwAAuHidt4CSnp6utLQ0paamerRv3bpVR48e9WhPTExU7dq1tX79+grHysjIUFhYmHuJi4s7X2UDAAALnJeAkpWVpW3btikjI6PcutzcXAUGBio8PNyj3eVyKTc3t8Lxhg8froKCAveyZ8+e81E2AACwRBVvD7hnzx498MADWr16tYKCgrwyptPplNPp9MpYAADAfl6fQdm6davy8/PVvHlzValSRVWqVNGaNWs0ZcoUValSRS6XSyUlJTp48KDH8/Ly8hQTE+PtcgAAQCXk9RmUTp066ZNPPvFoGzBggBITE/WPf/xDcXFxCggIUHZ2tnr27ClJ2rFjh7777julpKR4uxwAAFAJeT2ghISE6Morr/RoCw4OVlRUlLt94MCBGjp0qCIjIxUaGqr77rtPKSkpat26tbfLAQAAlZDXA8qZePbZZ+Xn56eePXuquLhYXbp00bRp03xRCgAAsNAFCSjvvfeex+OgoCBlZmYqMzPzQrw8AACoZHwygwIAuPDih73p6xJ8Ytf4NF+XgHPAlwUCAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA7fZgyv+LN+S6rEN6UCwPnADAoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6VXxdAPBnFj/sTV+X4BO7xqf5ugQAlmMGBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOt4PaBkZGSoZcuWCgkJUY0aNdSjRw/t2LHDo8+RI0eUnp6uqKgoVa9eXT179lReXp63SwEAAJWU1wPKmjVrlJ6erg0bNmj16tU6evSoOnfurKKiInefBx98UG+88YYWLVqkNWvWaN++fbrpppu8XQoAAKikqnh7wJUrV3o8njNnjmrUqKGtW7fqmmuuUUFBgV588UXNnz9fHTt2lCTNnj1bDRo00IYNG9S6dWtvlwQAACqZ834OSkFBgSQpMjJSkrR161YdPXpUqamp7j6JiYmqXbu21q9fX+EYxcXFKiws9FgAAMDF67wGlNLSUg0ZMkRt2rTRlVdeKUnKzc1VYGCgwsPDPfq6XC7l5uZWOE5GRobCwsLcS1xc3PksGwAA+Nh5DSjp6en69NNPlZWV9YfGGT58uAoKCtzLnj17vFQhAACwkdfPQSkzePBgLV++XGvXrlWtWrXc7TExMSopKdHBgwc9ZlHy8vIUExNT4VhOp1NOp/N8lQoAACzj9RkUY4wGDx6spUuX6p133lFCQoLH+qSkJAUEBCg7O9vdtmPHDn333XdKSUnxdjkAAKAS8voMSnp6uubPn6/XX39dISEh7vNKwsLCVLVqVYWFhWngwIEaOnSoIiMjFRoaqvvuu08pKSlcwQMAACSdh4Ayffp0SVKHDh082mfPnq3+/ftLkp599ln5+fmpZ8+eKi4uVpcuXTRt2jRvlwIAACoprwcUY8xp+wQFBSkzM1OZmZnefnkAAHAR4Lt4AACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdnwaUzMxMxcfHKygoSMnJydq0aZMvywEAAJbwWUB59dVXNXToUI0aNUrbtm1TkyZN1KVLF+Xn5/uqJAAAYAmfBZRJkybprrvu0oABA9SwYUPNmDFD1apV00svveSrkgAAgCWq+OJFS0pKtHXrVg0fPtzd5ufnp9TUVK1fv75c/+LiYhUXF7sfFxQUSJIKCwvPS32lxb+cl3Ft90f25591n0nst3Nxvn52cWp83s7en3WfSefn57RsTGPMafv6JKAcOHBAx44dk8vl8mh3uVz68ssvy/XPyMjQmDFjyrXHxcWdtxr/jMIm+7qCyon9dvbYZ7iQ+Lydm/O53w4dOqSwsLBT9vFJQDlbw4cP19ChQ92PS0tL9dNPPykqKkoOh8OHlXlXYWGh4uLitGfPHoWGhvq6nEqBfXZu2G/nhv12bthvZ+9i3WfGGB06dEixsbGn7euTgHLJJZfI399feXl5Hu15eXmKiYkp19/pdMrpdHq0hYeHn88SfSo0NPSi+kBeCOyzc8N+Ozfst3PDfjt7F+M+O93MSRmfnCQbGBiopKQkZWdnu9tKS0uVnZ2tlJQUX5QEAAAs4rNDPEOHDlW/fv3UokULtWrVSpMnT1ZRUZEGDBjgq5IAAIAlfBZQevXqpR9++EEjR45Ubm6umjZtqpUrV5Y7cfbPxOl0atSoUeUOZ+Hk2Gfnhv12bthv54b9dvbYZ5LDnMm1PgAAABcQ38UDAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBRLZGZmKj4+XkFBQUpOTtamTZt8XZL11q5dq27duik2NlYOh0PLli3zdUnWy8jIUMuWLRUSEqIaNWqoR48e2rFjh6/Lst706dPVuHFj9109U1JStGLFCl+XVamMHz9eDodDQ4YM8XUpVhs9erQcDofHkpiY6OuyfIKAYoFXX31VQ4cO1ahRo7Rt2zY1adJEXbp0UX5+vq9Ls1pRUZGaNGmizMxMX5dSaaxZs0bp6enasGGDVq9eraNHj6pz584qKirydWlWq1WrlsaPH6+tW7dqy5Yt6tixo7p3767PPvvM16VVCps3b9bMmTPVuHFjX5dSKTRq1Ej79+93L++//76vS/IJ7oNigeTkZLVs2VJTp06VdPy2/3Fxcbrvvvs0bNgwH1dXOTgcDi1dulQ9evTwdSmVyg8//KAaNWpozZo1uuaaa3xdTqUSGRmpp556SgMHDvR1KVY7fPiwmjdvrmnTpmns2LFq2rSpJk+e7OuyrDV69GgtW7ZM27dv93UpPscMio+VlJRo69atSk1Ndbf5+fkpNTVV69ev92Fl+DMoKCiQdPyXLc7MsWPHlJWVpaKiIr477Aykp6crLS3N4/84nNrOnTsVGxurunXrqm/fvvruu+98XZJP+OxW9zjuwIEDOnbsWLlb/LtcLn355Zc+qgp/BqWlpRoyZIjatGmjK6+80tflWO+TTz5RSkqKjhw5ourVq2vp0qVq2LChr8uyWlZWlrZt26bNmzf7upRKIzk5WXPmzNEVV1yh/fv3a8yYMWrXrp0+/fRThYSE+Lq8C4qAAvxJpaen69NPP/3THt8+W1dccYW2b9+ugoICLV68WP369dOaNWsIKSexZ88ePfDAA1q9erWCgoJ8XU6l0bVrV/e/GzdurOTkZNWpU0cLFy780x1OJKD42CWXXCJ/f3/l5eV5tOfl5SkmJsZHVeFiN3jwYC1fvlxr165VrVq1fF1OpRAYGKjLLrtMkpSUlKTNmzfrueee08yZM31cmZ22bt2q/Px8NW/e3N127NgxrV27VlOnTlVxcbH8/f19WGHlEB4erssvv1xff/21r0u54DgHxccCAwOVlJSk7Oxsd1tpaamys7M5vg2vM8Zo8ODBWrp0qd555x0lJCT4uqRKq7S0VMXFxb4uw1qdOnXSJ598ou3bt7uXFi1aqG/fvtq+fTvh5AwdPnxY33zzjWrWrOnrUi44ZlAsMHToUPXr108tWrRQq1atNHnyZBUVFWnAgAG+Ls1qhw8f9virIicnR9u3b1dkZKRq167tw8rslZ6ervnz5+v1119XSEiIcnNzJUlhYWGqWrWqj6uz1/Dhw9W1a1fVrl1bhw4d0vz58/Xee+9p1apVvi7NWiEhIeXObQoODlZUVBTnPJ3Cww8/rG7duqlOnTrat2+fRo0aJX9/f912222+Lu2CI6BYoFevXvrhhx80cuRI5ebmqmnTplq5cmW5E2fhacuWLbr22mvdj4cOHSpJ6tevn+bMmeOjquw2ffp0SVKHDh082mfPnq3+/ftf+IIqifz8fN15553av3+/wsLC1LhxY61atUp/+ctffF0aLjLff/+9brvtNv3444+Kjo5W27ZttWHDBkVHR/u6tAuO+6AAAADrcA4KAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKzz/wB5PMN8PihzSQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "score_list = [x.scores[\"answer_similarity\"] for x in scores.run_data]\n",
    "value_counts = pd.Series(score_list).value_counts()\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(list(value_counts.index), list(value_counts))\n",
    "ax.set_title(\"Answer Similarity Score Value Counts\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f42f83e7-bab1-41d5-b7f9-710e34f0a7ea",
   "metadata": {},
   "source": [
    "As 0 is the most common score, there is much room for improvement. This makes sense, as we are using the default parameters. We could imrpove these results by tuning the many possible RAG parameters to optimize this score."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (tvalmetrics_test_env)",
   "language": "python",
   "name": "tvalmetrics_test_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
